Repository navigation
fix(dev): honor request max_concurrency - #2730
Merged
Luca Forstner (lforst) merged 1 commit intoNov 5, 2026
Merged
Conversation
The playground and "+ Experiment" dialog now send max_concurrency to remote eval dev servers (braintrust#24075), but evalBodySchema dropped it, so lowering the slider had no effect on remote evals. Parse max_concurrency from the eval request and pass it to Eval(), capped at the evaluator's own maxConcurrency. The request can lower the limit but never raise it, so the playground's default of 10 cannot exceed a limit set in code. Refs: COR-288 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Luca Forstner (lforst)
approved these changes
Nov 5, 2026
Luca Forstner (lforst)
deleted the
fix/devserver-request-max-concurrency
branch
November 5, 2026 12:11
Abhijeet Prasad (AbhiPrasad)
pushed a commit
to braintrustdata/braintrust-sdk-python
that referenced
this pull request
Nov 5, 2026
Follow-up to braintrustdata/braintrust#24075, which made the playground and "+ Experiment" dialog send `max_concurrency` to remote eval dev servers. This PR makes the dev server honor it. Companion JS SDK PR: braintrustdata/braintrust-sdk-javascript#2730 ### Context The playground **Max concurrency** slider and the "+ Experiment" **Max concurrency** field had no effect on remote eval tasks. braintrustdata/braintrust#24075 (merged) made the UI send `max_concurrency` in the `POST /eval` request to the dev server, but `parse_eval_body` never read the field and `run_eval` built the `EvalAsync` kwargs only from the registered evaluator, so it was ignored. The value set on the Eval in code always won. A customer lowered the slider to avoid Azure 429s and saw no change (Pylon 20534, COR-288). ### Description Parse `max_concurrency` from the eval request and pass it to `EvalAsync`. **The request value can lower the evaluator's limit but never raise it** (`min(request, evaluator)`; if the evaluator sets no limit, the request value applies). This is deliberate: the playground always sends a value and the slider defaults to 10, so letting the request override outright would silently raise every code-set limit below 10 (e.g. `max_concurrency=5` → 10) as soon as users upgrade the SDK. The trade-off is that the UI can't push concurrency *above* what's in code. Other details: - `null` or a missing value keeps the evaluator's limit, which also covers older UIs that don't send the field. - Fractional values are rounded down (the "+ Experiment" field is a free number input validated only as `>= 1`). Values below 1 or non-numbers are rejected with a 400. ### Testing - New tests cover: request lowers the limit, request can't raise it, request applies when the evaluator has no limit, null or omitted request keeps the evaluator's limit, rounding down, and invalid values. They fail without the fix and pass with it. - `nox -s test_cli` passes (43 passed, 2 skipped); `make lint` is clean. - End to end: ran the real dev server (`eval --dev`) from this branch against a Braintrust org, with a 12-row eval whose task records how many calls are in flight, and sent `POST /eval` requests shaped like the playground's. Same run against the released Python 0.49.2 (PyPI) for comparison: | Evaluator limit in code | Request `max_concurrency` | Expected peak | This branch | Released | |---|---|---|---|---| | 5 | 2 | 2 | **2** | 5 | | 5 | 10 | 5 | 5 | 5 | | 5 | `null` | 5 | 5 | 5 | | 5 | omitted | 5 | 5 | 5 | | 5 | 2.5 | 2 | **2** | 5 | | none | 3 | 3 | **3** | 12 | | none | omitted | 12 | 12 | 12 | Refs: COR-288 Co-authored-by: Paul Tancre <paultancre@p-tancre.home> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to https://github.com/braintrustdata/braintrust/pull/24075, which made the playground and "+ Experiment" dialog send
max_concurrencyto remote eval dev servers. This PR makes the dev server honor it.Companion Python SDK PR: braintrustdata/braintrust-sdk-python#1072
Context
The playground Max concurrency slider and the "+ Experiment" Max concurrency field had no effect on remote eval tasks. braintrustdata/braintrust#24075 (merged) made the UI send
max_concurrencyin thePOST /evalrequest to the dev server, butevalBodySchemastripped the field and the/evalhandler spreads...evaluatorintoEval(), so it was ignored. The value set on the Eval in code always won. A customer lowered the slider to avoid Azure 429s and saw no change (Pylon 20534, COR-288).Description
Parse
max_concurrencyfrom the eval request and pass it toEval()asmaxConcurrency.The request value can lower the evaluator's limit but never raise it (
min(request, evaluator); if the evaluator sets no limit, the request value applies). This is deliberate: the playground always sends a value and the slider defaults to 10, so letting the request override outright would silently raise every code-set limit below 10 (e.g.max_concurrency=5→ 10) as soon as users upgrade the SDK. The trade-off is that the UI can't push concurrency above what's in code.Other details:
nullor a missing value keeps the evaluator's limit, which also covers older UIs that don't send the field.>= 1). Values below 1 or non-numbers are rejected with a 400.Testing
vitest run dev/server.test.ts(13 passed),pnpm run check:typings, and prettier are clean. Changeset included (patch).eval --dev) from this branch against a Braintrust org, with a 12-row eval whose task records how many calls are in flight, and sentPOST /evalrequests shaped like the playground's. Same run against the released JS 3.42.0 (npm) for comparison:max_concurrencynullRefs: COR-288